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(54)[Title of the Invention] 

INFORMATION PROCESSING APPARATUS 

(57)[Abstract] 

[Objective] To provide an information processing apparatus which can process, present 
information that a user requires, out of various and plenty of information, by an 
easy-to-use operation, efficiently. 

[Configuration] It is characterized in that, on a parallel with.presentation of information, 
an evaluated value regarding the presentation information, which is inputted by a user, is 
stored, and as to high and low of this evaluated value inputted, since it exceeds a 
predetermined range, there is a thing which is changed within the range and recorded, 
and at that time, the evaluated value is automatically measured and amended, or the 
evaluated value recorded is modified by an instruction of a user, and furthermore, in 
accordance with a numerical value of high and low of the evaluated value 
amended/modified in this manner, summary presentation of the information is carried 
out. 

1 1 6 SOUND PRESENTING SECTION 
1 1 5 SOUND PREPARING SECTION 
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[Claims] 

[Claim 1] An information processing apparatus which presents at least one of 

image information and sound information to a user, characterized by being equipped with 
means which inputs, modifies a desired evaluated value by a predetermined 
information unit, to the presentation information, in the same screen as a screen of 
preventing the information, 

[Claim 2] The information processing apparatus as set forth in Claim 1 3 

characterized in that the inputting/modifying means includes means which inputs, 
modifies plural kinds of evaluated values to the same information. 

[Claim 3] The information processing apparatus as set forth in Claim 1 or 2, 

characterized by being further equipped with means which selects any one evaluated 
value, out of the plural kinds of evaluated values, and means which extracts only 
information, in which the selected evaluated value is a predetermined value or more, out 
of the information, and presents it. 

[Claim 4] The information processing apparatus as set forth in Claim 3, 

characterized in that the selecting means includes means which selects two or more 
evaluated values 3 and furthermore, means, which calculates a new evaluated value by a 
predetermined numerical calculation, from evaluated values, on the occasion that two or 
more evaluated values are selected by the selecting means, is provided, and the 
presenting means carries out extraction on the basis of this evaluated value-which is 
newly calculated, 

[Claim 5] The information processing apparatus as set forth in Claim 1, 2, 3 or 4, 

characterized in that the evaluated value is a value which is set within a predetermined 
range, and after either one of a minimum value of a maximum value of the evaluated 
value has been already set up, on the occasion that the need arises to associate an 
evaluation which is less than this minimum value or an evaluation which is more than the 
maximum value with the presentation information, the inputting/modifying means 
includes means which tentatively inputs, modifies an evaluated value showing the 
evaluation which is less than the minimum value or the evaluation which is more than the 
maximum value, and furthermore, when the evaluated value, which shows the evaluation 
that is less than the minimum value or the evaluation that is more than the maximum 

4 
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value, is inputted, modified tentatively, this evaluated value is replaced with the minimum 
value or the maximum value, and characterized by being equipped with means which 
smooth-amends an already associated other evaluated value within a predetermined 
period, on the basis of this newly replaced minimum value or maximum value. 
[Claim 6] The information processing apparatus as set forth in claim 1, 2, 3, or 4, 

characterized by being further equipped with means which manages a ranking between 
information by the evaluated values, and corrects the evaluated value by operating a 
ranking relation of the information. 

[Claim 7] The information processing apparatus as set forth in Claim 1, 2, 3, 4, 5 

or 6, characterized by being further equipped with means which observes eye movement 
of a user on the occasion of inspecting the presentation information, means which 
sequentially calculates a speed component of eye movement from this observed eye 
movement, and means which obtains an evaluated value to the information from this 
calculated speed component. 
[Detailed Description of the Invention] 
[0001] 

[Industrial Field of Application] This invention relates to a multimedia mformation 
processing apparatus which displays, reproduces, and edits information such as images, 
sounds, and texts, in an integrated fashion. 

. [0002] 

[Prior Art] Performance improvement of a microprocessor, which is mounted on a 
personal computer (PC) etc, is remarkable, and it is well known that environment where 
a lot of information can be processed by a popular type PC at a high speed is being put 
into place. In addition, an information transmission channel becomes of large capacity 
in the former, by development of memory devices etc. through optical communication 
technologies and communication control technologies, and magneto-optical disks and 
highly integrated circuits in recent years, and in the latter, infonnalion storage devices 
become of large capacity, and by the action of these matters, it is being brought to 
realization that it is possible to obtain a lot of information electronically by use of an 
inexpensive device in a home or office. 

[0003] In addition, in the same manner, it is possible to transmit information by a 

5 
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simple operation, and therefore, privilege of information transmission, which was 
restricted to broadcasting stations, newspaper publishing companies, etc. so far, prevails 
in ordinary uses, and furthermore, a lot of information is getting to come and go. This 
can be sensed also from popularization of current personal computer communications, 
electronic mails, electronic news etc. In addition, a terminal, which can exchange 
information in this way, is not limited to PC, but penetrated in all forms of devices, such 
as electronic notebooks, portable telephones and facsimiles. Then, types of information 
which they can handle are spread from only conventional textual information (text data) 
to sound/music information, still image information, and moving image information, and 
hs quality is also improved dramatically. 

[0004] However, with regard to a technique of information classification, there are 
many scenes in which a conventional method has been taken, and for example, if the 
world of electronic news is taken as an example, hundreds of news groups (= "bundle" 
with respect to each topic) exist, and several tens of postings per one day are made to 
each news group. Then, a user has to spend time more than a little, in order to find a 
news that he/she needs, or is interested in, out of them. This is because each news is 
handled by a layered structure of "news group -> news", which is very simple and a user 
can not define normally. Even in music tapes and video tapes, a starting position with 
respect to each music/program can be judged by detecting a silent portion, and an index 
signal of image recording start, and so on, but it is not possible to record information 
which indicates contents of the music/program, or an especially interested scene and 
portion, and therefore, there is no other way but that a user writes information, which 
shows them, on their labels etc. by oneself. 

[0005] Development of automating these information classifications, or a work of 
extracting a characteristic portion out of a plurality of information has been carried out 
extensively even now. However, many of them aim to substitute for human's judgment 
mechanism, and need advanced artificial intelligence, and it must be said, from a phase of 
development time, and also from a phase of cost, that cost is required too much at 
present. In addition, there is need for a user to instruct what he/she desires, to the 
suchlike automatic classification system, and it is hard to handle it for a user which does 
not have a clear vision, such as what he/she wants to see, and what-like information 

6 
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he/she wants to obtain. It is possible to assume that the suchlike unfamiliar user will 
increase drastically, just in such an era that information overflows. 

[0006] What is needed is rather information showing !, A place where it is felt as 
interesting (= interesting) is here", and it is not necessary needed to described its 
contents. For example, when an attractive actor appeared on only one scene in a movie, 
information showing who is that actor, whether the actor is a man or a woman, whether 
that object (photographic subject) is a human, is not always needed, and there is a lot of 
cases that a sufficient condition is satisfied in case of a device which indicates merely the 
scene. Nevertheless, in a flow of current information classification, a procedure of "a 
device examines information carefully — > it extracts a characteristic portion as a 
candidate — > it is checked out with request information which a user inputted — > 
classification/presentation are carried out" is used, and therefore, analysis of information 
contents has to be carried out carefully once 

[0007] On the one hand, at the present day that ordinary users are about to become 
information transmission sources, demand of information processing tools is growing. 
As conventional processing tools, there are a word processor for textual information, 
CAD (Computer Aided Design) and drawing software for graphical information, etc. 
However, it must be said that there are very few tool for the public with good operability, 
with respect to video and sound, among others, A user carries out "cut and paste" 
editing, by a work of connecting 2 units of decks, finding and reproducing a- place 
targeted through the use of one deck, and recording it through the use of the other deck. 
In order to cany out the "cut and paste" edit, 2 units of decks are required at minimum, 
and also an operation is troublesome, and therefore, an ordinary user tends to keep a 
distance from the editing work. Among editing methods which have been proposed at 
present, there is a thing which once stores video/sound information in a storage medium 
such as a memory and a hard disk of a computer, and processes information in an editing 
environment of a computer, but in order to do this, a large capacity storage medium is 
required, and in addition, cost is high, and it is "cut and paste'* editing at heart, and 
therefore, it is not possible to expect particular improvement of operability. 
[0008] As above, the feet that there is no simple and efficient information automatic 
classification mechanism and information processing means may become a large obstacle 

7 
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to future popularization of multimedia integrated environment. 
[0009] 

[Problem that the Invention is to Solve] As described above in detail, in a conventional 
information processing apparatus, there was no means which carries out 
classification/arrangement etc. for a wide variety of information efficiently beyond a type 
of information, and by reflecting a user's intention. On this account, a user has to find 
time and labor for a processing work of the information, and also in case of carrying this 
automatically as a device, there was such a drawback that it is not necessarily possible to 
cany out processing in an adapted fashion to individual users. In addition, 
conventionally, there was such a drawback that a processing method of information is 
troublesome, and in order to realize it, high cost is easily realized. 

[0010] The present invention is a thing which was made in view of the 
above-mentioned actual condition, and aims to provide an information processing 
apparatus which can process, present information that a user requires, out of various and 
plenty of information, by an easy-to-use operation, efficiently. 
[0011] 

[Means for Solving the Problem] The present invention is an information processing 
apparatus which presents at least one of image information and sound information to a 
user, and is characterized by being equipped with means which inputs, modifies a desired 
evaluated value by .a predetermined information unit, to the presentation -information, in. 
the same screen as a screen of preventing the information. 

[0012] In addition, the present invention is characterized in that the 
inputting/modifying means includes means which inputs, modifies plural lands of 
evaluated values to the same information. 

[0013] In addition, the present invention is characterized by being further equipped 
with means which selects any one evaluated value, out of the plural kinds of evaluated 
values, and means which extracts only information, in which the selected evaluated value 
is a predetermined value or more, out of the information, and presents it. 
[0014] In addition, the present invention is characterized in that the selecting means 
includes means which selects two or more evaluated values, and furthermore, means, 
which calculates a new evaluated value by a predetermined numerical calculation, from 
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evaluated values, on the occasion that two or more evaluated values are selected by the 
selecting means, is provided, and the presenting means carries out extraction on the basis 
of this evaluated value which is newly calculated. 

[0015] In addition, the present invention is characterized in that the evaluated value is a 
value which is set within a predetermined range, and after either one of a minimum value 
of a maximum vaJue of the evaluated value has been already set up, on the occasion that 
the need arises to associate an evaluation which is less than this minimum value or an 
evaluation which is more than the maximum value with the presentation information, the 
inputting/modifying means includes means which tentatively inputs, modifies an 
evaluated value showing the evaluation which is less than the minimum value or the 
evaluation which is more than the maximum value, and furthermore, when the evaluated 
value, which shows the evaluation that is less than the minimum value or the evaluation 
that is more than the maximum value, is inputted, modified tentatively, this evaluated 
value is replaced with the minimum value or the maximum value, and characterized by 
being equipped with means which smooth-amends an already associated other evaluated 
value within a predetermined period, on the basis of this newly replaced minimum value 
or maximum value. 

[0016] In addition, the present invention is characterized by being further equipped 
with means which manages a ranking between information by the evaluated values, and 
.corrects the evaluated, value by operating a ranking relation of the. information. 
[0017] In addition, the present invention is characterized by being further equipped 
with means which observes eye movement of a user on the occasion of inspecting the 
presentation information, means which sequentially calculates a speed component of eye 
movement from this observed eye movement, and means which obtains an evaluated 
value to the information from this calculated speed component. 

[0018] That is, an information processing apparatus, which relates to the present 
invention, is characterized by including means in which a user sequentially inputs, 
modifies an evaluated value of for example 'important/unimportant' * etc., to information 
presented, in the same screen, along the line of information presentation, means which 
automatically modifies to an input which has no other choice but exceeding a 
predetermined range, out of the evaluated value inputted, means which corrects the 
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evaluated value inputted, on the basis of a ranking operation of a user, and means which 
extracts and presents a portion to be presented, out of original information, on the basis 
of the evaluated value modified/changed, 
[0019] 

[Operation] According to an information processing apparatus which relates to the 
present invention, inputting/modifying processing is carried out by associating an 
evaluated value of for example 'important/unimportant" etc. which a user has in one's 
arm, to information, by an independent standard, with information which is presented, in 
the same screen. On this account, it is possible to obtain an information value to a user, 
without using a lot of advanced information processing such as image analysis and voice 
recognition. In addition, the evaluated value, which was inputted in this way, is proper 
to a user for itself, and it can be expected that it is adapted to a user more, even if it is 
compared to an information value automatically judged by an apparatus only from 
information contents. 

[0020] In addition, an information processing apparatus, which relates to the present 
invention, modifies the above-mentioned inputted evaluated value of for example, 
"important/unimportant" etc. in accordance with an operation of a user. On this 
account, even in case that a user's standard of a value is changed during a period of 
information presentation, it can be expected to obtain an appropriate ranking of 

information values 

[0021] Furthermore, as described above, in information to which an information value 
was given by a user, it is all right if processing of extraction etc. of an important portion 
is carried out on the basis of the above-described evaluated value of 
"important/unimportant", and its mechanism is simple. In addition, also with respect to 
storage of such summary information that only an important portion was extracted from 
original information, it is all right if only a numerical value of the evaluated value is 
stored, and there is no need to store summary information in the complete form (i.e., on 
the occasion of preparing a 30 minute summary from 2 hour images, a storage location 
for 2 hours and 30 minutes was required in the past, but in a method of the present 
invention, it is all right if 2 hour original images and a numerical value of an evaluated 
value are simply stored), and therefore, saving of an amount consumed of a storage 
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medium can be expected. 
[0022] 

[Embodiments] Hereinafter, embodiments of the present invention will be explained 
with reference to drawings. 

[0023] (First Embodiment) Fig. 1 is a block diagram which shows a configuration of an 
information processing apparatus relating to a first embodiment. 

[0024] In the same embodiment, as a method of inputting a user's evaluated value 
about 'Important/unimportant", a case of detecting a user's sight line and automatically 
judging/deciding the evaluated value (hereinafter, referred to as an importance level) 
from an appearance of movement of the sight point will be explained. Meanwhile, as to 
a method of calculating an importance level from movement of a sight line will be 
described in detail in a third embodiment 

[0025] Information, which is read out from a recording medium 107 through a reading 
control section 105, is displayed on a presenting section 1076 9 but a user's sight line at 
that time is position-detected by a detector 101. In a judgment section 102, the sight 
information is recorded for a given length of time, and processed. This processing is 
carried out for example in such a manner that user's visual points for 5 seconds are 
detected every 1/30 second, and every time 150 pieces of data are collected, importance 
level estimation calculation is carried out, and so on. In the judgment section 102, from 
incoming visual point data, an important level in the recording. time is calculated, - In this 
example, it is carried out in such a manner that an importance level for previous 5 
seconds is outputted every 5 seconds. Here, importance level calculation is not limited 
to a case that it is carried out in the same cycle as a recording interval. For example, by 
interposing a process of carrying out importance level calculation by adding visual point 
data for 2.5 seconds on a past side among 5 second data which was sent this time, and 
data for 2.5 seconds on a current side among 5 second data which was sent this time, an 
importance level output at 2.5 second intervals becomes possible. As a matter of 
course, it is also possible to further shorten an output interval by the same method. 
[0026] Importance levels, which are sent by the judgment section 102 at given output 
intervals, are stored in an importance level information temporary storing section 110 
over entire information presented, and recorded on the recording medium 107. The 
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entire information presented, here, a zone etc. which correspond to one packet of stories 
such as one program, one movie, and one trip portion of photographic album, and 
normally, as to these things, a section has been already clarified by a conventional 
method (index signal of VTR, one volume of movie software, index of album, or one file 
of computer data, etc.). It is all right even if the recording medium 107 is the same 
medium as one in which information presented was recorded, or a different medium. 
For example, it. is all right even if importance level information regarding image 
information, which was read out from an magneto-optical disk and presented, is stored in 
an original magneto-optical disk, or a memory IC for use in importance level storage is 
used. Alternatively, it is all right if importance level information regarding information, 
which was read out from a remote recording device connected by a communication 
network, and presented, (e.g., video-on-demand, etc) is returned to an original 
recording device through a network and recorded, and it is all right if it is recorded on a 
computer side which a user is utilizing as a terminal- In this example, it was explained 
about a case of temporarily storing importance level information in the importance level 
information temporary storing section 110 over entire information presented, but it is all 
right if the judgment section 102 carries out importance judgment one after another even 
during a period of presenting information, and under such a condition that original 
information presentation and importance level information recording can be carried out 
simultaneously on the.recording medium 107, in^ortance.level.infomiation..is-recorded- 
on the recording medium 107 on a parallel with information presentation. 
[0027] Next, on the occasion of desiring to change importance information, the 
recorded importance level is read out from the recording medium 107, and modification 
is applied with regard to entire information presented. As to this modification, there are 
one which an apparatus carries out automatically by a modification calculating section 
1 04, and one which is carried out in accordance with an operation of a user. Firstly, the 
modification, which is carried out automatically by the modification calculating section 
104, will be explained. 

[0028] In case of inputting importance levels one after another, a level, which is 
inputted later in time in terms of time, is of a lower freedom degree. For example, it is 
assumed that, in case of inputting importance levels from 0 up to 100 over seeing a 2 
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hour movie, a level 100 was given, with having the highest interest al the time of 55 
minute passage. However, after that, even if it is felt that the higher interest than ever 
arose at the time of 1 hour and 40 minute passage, it is not possible to give the highest 
level which exceeds the previous one, any more. Then, by the following method, an 
actual highest level is estimated, considering that, in case that an input limit highest level 
was continued, it reached to a level which exceeds an input limit in reality. 
[0029] Fig.2 is a flow chart which shows algorithm of modifying an importance level in 
the information processing apparatus that relates to the same embodiment. 
[0030] This operation is practically carried out over exchanging importance level 
information between the modification calculating section 104 and the importance level 
information temporary storing section 110 in Fig-L In the following explanations, and 
Fig.2, DATA is the number of data to be modified, and tn is n-th (n starts from 0) 
sampling time, nad In is an importance level at time tn. In addition, (t ave,k, I ave,k) is 
a k-th coordinate at the time of smoothing (tn, In). T hold, I hold, n, k are temporary 
variables which are used only in this algorithm. I max is the permissible highest value of 
an importance level, and for example in case of trying to input a height of an importance 
level in a range from 0 to 100 3 1 max becomes 100. 

[0031] In the flow chart shown in Fig.2, it shows only such a portion that temporal 
transition of an importance level is smoothed. A smoothed level I ave has the same 
starting value and. ending value as. those of an original importance level, I as shown in a 
step S201 and a step S218. In addition, at the time of continuing such a situation that 
an importance level is the maximum value I max for 3 samples, assuming that a level 
exceeding I max could exist in this zone (hereinafter, referred to an overflow zone) (step 
S206, step S2] I), an entrance and an exit of the overflow zone are conformed with an 
original importance level (step S213 - step S217). In post-processing, the overflow 
zone is detected, and a virtual maximum level in the zone is estimated. In a zone other 
than that, a middle point of a maximum value and a minimum value is used as a 
coordinate point of a smoothed level (step S207, step S212). 

[0032] Next, by utilizing a smoothed level (t ave,k, I ave, k), a change of an 
importance level in the overflow zone is estimated, but this will be explained by use of 
Fig.3. 

13 



PAGE 45/116 * RCVD AT 611/2006 2:39:03 PM [Eastern Daylight Time] * SVR:USPTO-EFXRF-5/3 * DNIS:2738300 * CSID:216 579 6073 * DURATION (mm-ss):31-20 



06/01/06 13:45 F AX 216 579 6073 



PE ARNE GORDON 



[0033] In the overflow zone 301, at an entrance 302 and an exit 303 of the zone, it is 
estimated as a quadratic function or tertiary function having the same value > the same 
inclination as those of a change outside the zone (boundary condition). For example, 
assuming that times of the entrance 302, the exit 303 of the overflow zone 301 are ts, 
te 3 respectively, and inclination of a smoothed level at the entrance is A, and inclination 
at the exit is (-B), a formula of an importance level F (304) at time t estimated in the 
overflow zone 301 becomes 
[0034] 

[Mathematical Formula 1] 

r = (A-B)(t-ts) 3 /(te-ts) 2 - (2A-b)(t-ts) 2 /(te-ts) + A(t-ts) + Imax (1)' 

[0035] Next, on a continuous curve line 304 drawn in this way, a thing, which exists at 
original time ti, is substituted as new Ii. The above-mentioned process can be carried 
out one after another, as soon as necessary data is collected respectively, and therefore, 
it is processed on a parallel with an input of importance level, and at the time of input 
ending, it is possible to fix Ii in which, an overflow zone was estimated. 
[0036] In the meantime, it is likely that data becomes a value which is more than I max, 
in the overflow zone, by Ii fixed so far, and therefore, Ii is converted by use of the 
following formula. 
[0037] 

[Mathematical-Formula 2] . . • - . . . 

Fi = (Wmax-2Imax)AVmax 3 X Ii 3 + (3 Imax-2 Wmax) AVmax 2 X Ii 2 + Ii 
(2) 

Here, Wmax is the highest value in entire importance level data. In this regard, 
however, in case that Wmax does not exceed Imax, this conversion is not carried out. 
This conversion has such a feature that the more a value is, the stronger it is compressed, 
and therefore, an influence applied to data, which was preexistent in a range of Imax, is 
few. In addition, there is not such a case that a magnitude relation of original data is 
lost by this conversion. Furthermore, since estimation of the overflow zone is carried 
out on the basis of smoothed data, an influence of local steep/slow rising of 3i 
immediately before an entrance and immediately after an exit of the overflow zone, is 
few. 
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[0038] In the foregoing, modification in case that an importance level has reached to 
maximum in a range of a specified value was explained, but it is all right even if 
modification is carried out by the same method in case that an importance level has 
reached to minimum in a range of a specified value. 

[0039] Next, an example of a case off carrying out change/modification of an 
importance level by an operation of a user will be explained by use of FigA 
[0040] This operation is processed as described below in Fig.l. Firstly, a user gives 
an instruction of the information processing apparatus of the same embodiment, by using 
an input device such as a mouse, or the sight line detector which was used earlier, as a 
pointing device. This instruction is received by the operation interface 109, and the 
operation interface 109 sends a signal to an address generating section 112 so as to 
output a screen/sound change (with regard to this, it will be explained later) which 
corresponds to an input of a user. In the address generating section 1 12, a position of 
original information to be displayed is outputted as a storage location (address) on the 
recording medium 107, and sent to a reading control section 105. The reading control 
section 105 controls the recording medium 107 mechanically, and takes out original 
information from an appropriate position on the medium. Original information, which 
was taken out, is sent to an original information temporary storing section 113, and 
becomes a graphical user interface in a screen generating section 114, together with 
importance infonnation which has been already inputted, and presented to. a.user through 
an image presenting section 106 and a sound presenting section 116. 
[0041] On the one hand, a user modifies an importance level over taking a look at this 
screen, but a change of an importance level, which goes with this operation, is carried 
out through a modification calculating section 104. The importance level, which has 
been already inputted, is stored in the importance level information temporary storing 
section 110, and modified by the modification calculating section 104 by this operation, 
and returned to the importance level information temporary storing section 110 again. 
When modification is completed, temporarily stored importance level information is 
outputted to a recording control section 103, and recorded on the recording medium 107 
at an appropriate storage location specified by the address generating section 112. As a 
matter of course, it is all right even if importance level information is divided into several 
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pieces, and one of the divided pieces is stored in the importance level information 
temporary storing section 110, and an operation of modification -> return -» recording 
— > temporary storage of next divided piece is repeated. 
[0042] Here, going back to Fig. 4, actual processing will be described. 
[0043] After an importance level is inputted by the above-described method, the 
information processing apparatus of the present invention cites a maximum value, a 
minimum value of an importance level, and time that they were recorded respectively. 
Then, an image 402, in which a maximum value was recorded, is presented on a screen, 
in the order from a larger one of maximum values ((a) of Fig.4). The number of images 
to be presented here can be set arbitrarily, such as 3 sheets, 5 sheets, and 10 sheets (Fig.4 
shows an example of 3 sheets). In addition, it is all right even if the image 402 to be 
presented is not necessarily an image at such a moment that a maximum value was 
shown, and it is all right even if it is a first screen of a cut including that maximum value 
(during a period of two successive scene change), and in addition, it is all right even if it 
is configured so as to carry out reproduction over an entire cut including the maximum 
value, by clicking a presented screen by a mouse. 

[0044] From this presented screen 402, a user can know how this apparatus estimated 
an importance level of a user oneself. When there is uncomfortable feeling about the 
estimated importance level which was presented, a user instructs correction of a ranking 
to this apparatus^ by- reordering this image. For example, in case that -a user felt that a 
scene 403, which this apparatus estimated as a 2nd place, is in fact more "important" 
than a scene 404 which was estimated as a 1st place, a user carries out an instruction to 
change the screen 403 of 2nd place onto the screen of 1st place. An operation of 
reordering is carried out by the input device 10S in Fig.l, but an image, which is desired 
to be corrected (in the above example, the screen 403 of 2nd place), is selected by a 
keyboard, a button or a key pad of a remote controller, etc., and then, a location of a 
new ranking (position 405 above the image of 1st place) is selected. Alternatively, it is 
all right if the same operation is carried out by use of a mouse. In addition, it is all right 
even if the selection is carried out by utilizing the sight line detecting device 101, and by 
keeping a sharp eye on the screen, 

[0045] When correction of a ranking is instructed, this apparatus decides which value 

16 



PAGE 48/1 16 * RCVD AT 6/1/2006 2:39:03 PM [Eastern Daylight Time] * SVR;USPTO-EFXRF-5/3 * DNIS:2738300 * CSID:216 579 6073 * DURATION (mm-ss):31-20 



06/01/06 13:49 FAX 216 579 6073 



PEARNE GORDON 



g)049 



an importance level in the vicinity of a screen should be changed to. For example, in 
case that correction was made to an intermediate between a 2nd place and a 3rd place, it 
becomes an intermediate value etc. of an importance level of the 2nd place and an 
importance level of the 3rd place. Alternatively, in case that correction was made to a 
level which exceeds a 1st place, it becomes a value which is higher than an importance 
level of the 1st place, by the same as a difference of importance levels of the 1st place 
and the 2nd place. For example, at the time that a level of a 1st place is 80 and a level 
of a 2nd place is 78 and a level of a 3rd place is 70, if a portion of the 1st place is 
corrected between the 2nd place and the 3rd place, an importance level of a change 
destination of the 1st place portion becomes 74. In addition, in the same example, if a 
portion of the 2nd place is corrected above the 1 st place, an importance level of a change 
destination becomes 90 (2nd place « 78, which has become an object of movement, is 
not used for calculation, and therefore, it does not become 84). A result of this 
correction is displayed on a screen (eg., like (b) of Fig.4) once more, and a user can 
know a result of one's own correction. 

[0046] A change procedure, at the time that an importance level of a change 
destination was decided as above, will be hereinafter explained by use of Fig- 5. 
[0047] Firstly, in case of changing an importance level to a lower side ((a) of Fig. 5), 
sandwiching a maximum value 503 which is desired to change, a range between 
maximum values (504, 505). in its vicinity, is changed.. Assuming that a changed 
maximum value is Ip and an importance level of a change destination is I adj, a function, 
which changes linearly between I and I adj/Ip, is multiplied by an original importance 
level, in a range between maximum values (506, 507) in the vicinity of the maximum 
value which is desired to change, and minimum values (506, 507) in the vicinity (from 
504 up to 506, from 507 up to 505). In a zone sandwiched by the minimum values in 
the vicinity (from 506 up to 507), a constant value I adj/lp is multiplied, to generate a 
new importance level. What is shown on a lower part of (a) of Fig, 5 is a schematic 
shape of the function to be multiplied. On the one hand, in case of changing an 
importance level to an upper side ((b) of Fig.5), sandwiching a maximum value 508 
which is desired to change, a range between minimum values (509, 510) in its vicinity, is 
changed. When Ip, I adj are defined in the sama manner as a case of changing to a 
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lower side, a function, which changes linearly between I and I adj/Ip, is multiplied by an 
original importance level, in a range between minimum values (509, 510) in the vicinity, 
and the minimum values 508 to be changed. In the same manner, what is shown on a 
lower part of (b) of Fig. 5 is a schematic shape of the function to be multiplied. 
[0048] If, as a result of this change, an importance level exceeds a maximum value I 
max of a predetermined importance level, it is amended to a range to fit in I max, by the 
same method as one described above, by use of Fig. 3. It is all right even if this 
amendment is carried out after all ranking changes are completed, and it is carried out 
with respect to each ranking change according to need. 

[0049] Finally, a method of processing, presenting original information automatically, 
by use of an importance level inputted/amended as above, will be explained. 
[0050] In this case, in Fig.l, both of original information and importance level 
information are read out from the recording medium 107 through the reading control 
section 105, and stored in the original information temporary storing section 113, the 
importance level information temporary storing section 110, respectively, but a portion 
to be presented, out of the original information is selected by a presenting portion 
selecting section 111, on the basis of the importance level information of the importance 
leveJ information temporary storing section 110. This will be described in detail later. 
In the address generating section 112, it receives information regarding the selected 
portion from the presenling-portion selecting section 1 1 1, and outputs a. location-on the 
recording medium 107, where the information to be presented has been recorded, as an 
address. In the reading control section 105, it accesses to the recording medium 107 
selectively on the basis of this address information, and sends a portion of the original 
information to the original information temporary storing section 113. Subsequent 
processes are the same as those in the screen display method which was already 
explained. 

[0051] Also here, as a matter of course, it is all right even if the address generating 
section 112 controls a reading location from the original information temporary storing 
section 1 13, in lieu of controlling a reading location from the recording medium. In this 
case, normally, it is possible to expect improvement of a reading speed (under present 
circumstances, a semiconductor memory device, which is used for the original 
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information temporary storing section 113, is faster than an access speed of an 
magneto-optical disk etc. which are used for the recording medium 107). In the 
following, a process of carrying out selection of a portion of original information in the 
presenting portion selecting section 111 will be explained concretely. 
[0052] As to a method of processing/presenting which will be explained here, there are 
2 ways of a case of detecting a semantic section of information (scene change of moving 
images, textual unit and paragraph unit of sound/document, phrase of music — pattern of 
development which is configured by approximately 4-16 bars), by existing another 
method such as image recognition and speech recognition, and utilizing the section, and a 
case of not utilizing it. 

[0053] Firstly, a case of utilizing a section will be explained by use of Fig.6. 
[0054] A user inputs a desired temporal length to presentation information which is 
generated as a result of information processing. It is all right even if this is carried out 
by a thing such as a volume knob, and even if it is inputted directly as "- minute, - 
second" by use of a keyboard. That is, if it is a method of inputting a time length, any 
method, which is commonly used, is used. In this apparatus, a threshold value 601 is 
gradually reduced from a specific importance level maximum value I max 602, and a 
total of a time zone 604, which exceeds the threshold value 601, is calculated, in time 
transition 603 of an importance level. Until this calculation result exceeds presentation 

desiring time . inputted, a process of reducing the threshold value 6Q1. continues In this.. 

way, when the threshold value 60 1 immediately before exceeding presentation desiring 
time is decided, this apparatus presents only the time zone 604 which exceeds the 
threshold value 601, in series. Alternatively, there is a method of citing information at a 
moment of the maximum value 605 which is more than the threshold value of an 
importance level, on a screen, and when the screen is selected by a mouse or gaze etc., 
presenting a time zone which exceeds the threshold value including this maximum value. 
By this means, a user can selectively see only a portion where an importance level is 
especially high, among all time of information. 

[0055] Next, a case in which a semantic separation of information has been already 
known and it is utilized, will be explained by use of Fig.7. 

[0056] Firstly, a case in which information is moving images will be explained. 
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[0057] It is assumed that an importance level 701 is given according to moving images, 
and scene changes 702, which are semantic separations of moving images, have been 
known. It is assumed that a time zone between two scene changes 702 which are 
adjacent in terms of time, is called as cut 703. In one cut 703, an importance level is 
accorded with a maximum value 704 in that cut 703, By doing this, it is possible to 
utilize the same method as that of the case of not utilizing a section of information. 
[0058] That is, a threshold value 705 is gradually reduced, to obtain the threshold 
value 705 immediately before a total of a time zone exceeding the threshold value 705, 
exceeds presentation desiring time which was set up. At this time, by presenting a time 
zone exceeding the threshold value 705, i.e., cuts one after another, it is possible to 
prepare a summary by cut unit. 

[0059] In addition, it is also possible to carry out the same cut selection as the 
above-mentioned one, by according an importance level in the cut 703, with such 
"importance level density 3 ' 705 that an integral value of an importance level in the cut 
703 is divided by a length of the cut 703, in lieu of using a maximum value in that cut. 
[0060] A difference of meanings possessed by these two selecting methods is that the 
former (an importance level in the cut 703 is the maximum value 704) evaluates the cut 
703 by strength of momentary maximum impression, whereas the latter (an importance 
level in the cut 703 is an average value 706 in the cut 703) evaluate by strength of 

impression given.by the entire cut 703 . . _ 

[0061] In the foregoing, an example of an operation of the address generating section 
1 12 on the occasion of carrying out "sequential present ation" will be explained by use of 
Fig.8. 

[0062] Among line graphs in Fig.8, a right side one ((b)) is such a thing that a graph of 
Fig.6 was rotated by 90 degree. A portion, which was selected by the method 
explained through the use of Fig. 6, is shown by hatching. In the suchlike case, at the 
time of sequentially presenting a selected portion, the address generating section 112 
outputs an address which is shown on a left side ((a)) of Fig.8 (in this regard, however, 
this is a case that time passage of original information was recorded linearly on the 
recording medium 107. In case that original information is recorded in an arrangement 
having no casual relation to time passage, a vertical axis of the graph on the left side of 

20 



PAGE 52/1 16 ' RCVD AT 6/1/2006 2:39:03 PM [Eastern Daylight Time] * SVR:USPTO-EFXRF-5/3 < DNIS:2738300 * CSID:216 579 6073 * DURATION (mm-ss):31-20 



06/01/06 13:50 FAI 216 579 6073 



PEARNE GORDON 



Fig. 8 shows a time position to be presented, in lieu of an address). That is, on the 
occasion of summary presentation, reproduction is earned out in a selected time zone at 
a normal operation speed, and a location, which is not so ? is skipped. An operation is 
the same even in case that a selected portion is of cut unit as described above. 
[0063] As to cuts which were selected as described above, in addition to a method of 
sequentially presenting the selected cuts from a front one in terms of time, there is a 
method of presenting from one having a higher importance level of a cut. In addition to 
that there is also a method in which only a front screen of the selected cuts is cited on a 
screen as a small screen, and when a user selects the small window by a mouse, gaze etc., 
a moving image of that cut is presented. Here, a screen, which represents a cut, does 
not necessarily have to be a front screen of a cut, and it is all right if a screen at such a 
moment that the highest importance level was shown in that cut is presented. 
Alternatively, it is all right even if only a location to be presented is selected and decided 
by a method described above, and it is configured so as to be able to freely change a 
speed and a direction of presentation time of respective zones, a ranking etc. by an 
operation of a user. When this is done, for example in case that still images, which are 
presented by 30 sheets every 10 seconds, are original information, it becomes possible 
to select only 10 sheets of images which are especially important, by use of a threshold 
value of an importance level, and to see one sheet, one sheet among the 10 sheets, only 
for time that one likes. . - - 

[0064] Next, an example of a case in which information is music (tune) and a 
separation of information has been known, will be explained by use of fig. 9. 
[0065] As appeared in Fig.9, a tune 801 shown in the example is composed of 10 
phrases. A portion, to which a dash (*) is attached, is modification of a portion of the 
same sign and with no dash. A is a portion normally called as lead-in (introduction), 
and D is a portion normally called as interlude, and F is a portion normally called as 
postlude (ending), and portions of songs are B'C (1st = 1st chorus), and B'CEC'(2nd = 
2nd chorus). By substituting phrases of A, B, etc. with cuts 703 in the explanation by 
use of Fig.7 ? the same processing is carried out, and if a phrase 802 with a high 
importance level is extracted, it is possible to listen only the most impressive portion, 
without listening entirety of the tune. In case of looking for a target tune, if only 
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summaries of tunes, which were prepared in this way, are listened one after another, it is 
possible to save time until reaching to a portion stuck out in one's mind, and therefore, it 
can be expected that time required for search is shortened. 

[0066] (Second Embodiment) Next, as a second embodiment, processing which a user 
carries out, and display at a presenting section, in an information processing apparatus of 
the present invention, will be explained in detail by use of figures. 

[0067] In the same embodiment, it is assumed that a mouse is used as a pointing device 
for input, but in the information processing apparatus which relates to the present 
invention, it is possible to use any pointing device. 

[0068] Firstly, an operation of a user in case of carrying out presentation of moving 
images by utilizing an importance level, and display at a presenting section, will be 
explained. 

[0069] When a user presents moving images, display as in for example, Fig. 10 or 1 1, is 
carried out on the display section. In Fig. 10, a moving image to be displayed is 
displayed on a large scale, and on the one hand, in Fig. 11, an importance level, which is 
given to a moving image, is clearly shown in the same screen. It is designed in such a 
manner that these display methods can be selected by a user depending on an intended 
use. Here, 1000 designates a lever for adjusting a threshold value of an importance 
level at the time of carrying out display of moving images through the use of an 
importance level, and if the lever is lifted up, only a portion of moving images .with a high .. 
importance level is presented, and inversely, if it is lifted down, also a portion with a low 
importance level is presented together. This is the same as one explained by use of 
Figs. 6 and 7 in the first embodiment. 

[0070] 1001 designates a presentation time display window, and is a thing which 
displays how much time, a total of portions, to which an importance level exceeding a 
threshold value was given in one moving image, becomes, when a threshold vale of an 
importance level was set up by the level 1000. In an example of this figure, it has 
become 12 minutes and 41 seconds. When a user wishes to complete viewing of rough 
contents of moving images within a certain period of time, it is all right if the user adjusts 
the level 1000 over looking at time in the window 1001. As a matter of course, as 
explained in the first embodiment, it is also possible to set up a threshold value of an 
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importance level, by directly punching time in the window 1001. On this occasion, a 
cursor is pointed to the window 1001, and clicked, and then, time is inputted from an 
input device such as a keyboard. If setting to desired time is completed, it is possible to 
see only portions with high importance level out of moving images, within a period of 
desired time, by pushing a "It will be reproduced from the beginning" button, out of 
operation buttons of 1008. On this occasion, it is possible to carry out presentation of 
moving images by the same procedures, in either one of a case of utilizing a semantic 
meaning of information which was explained in the first embodiment, and a case of not 
utilizing it. 

[0071] 1 103 designates an importance level display window, and in this, an importance 
level. 1004, which was given according to a time axis of moving images, is displayed. In 
addition, 1009 shows a L level of a threshold value which was set up by the importance 
level threshold value setting lever 1000. In the window 1003, a portion of entire 
moving images is displayed, and as to which portion is in entire moving images, it is 
displayed in a window of 1006. In an example of Fig.ll, entire moving images last 65 
minutes, whereas, what is displayed in the importance level setting window, 
approximately 15 minutes in a range from approximately 2 minutes up to 35 minutes. A 
user moves the lever 100 over looking at an importance level displayed in the window 
1003, and thereby, can adjust a threshold value of an importance level. For example, 

when he/she. is very, much interested in vicinity of a frame currently, displayed, out. of. . - 

moving images, and felt that he/she wishes to see its periphery more, it is possible to 
carry out reproduction of moving images after a threshold value of an importance level 
is lowered temporarily, and return the threshold value of the importance level to an 
original level again, once he/she finished seeing a certain interesting portion. 
[0072] The above operation can be also carried out to a group of still images and 
sounds, but not only to moving images. A group of still images, here, is a bunch of still 
images having any ranking. For example, photographs, which were taken at the time of 
travel, is an assembly of still images having a shooting order, and therefore, they are the 
group of still images which is spoken here. On the occasion of presenting a group of 
still images, a horizontal axis of the window 1003 represent a ranking of what number in 
still images, but not time. In addition, as to sounds having no images, it is all right if it 
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is designed so as to display wave forms of sounds in a display window of 1002, or 
display information of ON/OFF of specific sound in a bar chart form. In case of wave 
forms, centering around time which corresponds to a position of a cursor on 1003, wave 
forms over time in the vicinity are displayed. In case of displaying ON/OFF information 
of a sound, display of a brief moment status which corresponds to a position of a mouse 
cursor, or in the same manner as the case of wave forms, display of ON/OFF information 
over time in the vicinity, is carried out. 

[0073] In addition, in either case of moving images, a group of still images, and sounds, 
it is all right if it is designed in such a manner that a user can freely select a reproducing 
speed, or reproducing time, on the occasion of reproducing a portion selected by a 
threshold value of an importance level set by a user. For example, in case of moving 
images and sounds, they are displayed at a normal reproducing speed, and in addition, 
they are produced by selecting one from a method of presenting by changing a 
reproducing speed only by a scale factor determined by an importance level of that 
portion, and a method of controlling etc. a reproducing speed so as to display one 
scene (in case of sound, a section such as 1 bar line) sectioned by cuts, for the same time. 
In case of a group of still images, it is possible to select one from a method of not 
displaying a next still image until there is an instruction of a user, a method of displaying 
about 1 sheet of a still image only for a given length of time, or a method of displaying 

only for time determined by an importance level, and so on. 

[0074] So far, explanation was carried out on the premise of such a matter that only 
one type of an importance level is given to one moving image, but it is possible to give a 
plurality of importance levels. For example, it is possible to prepare an importance level 
with respect to each user, to one moving image and sound, a group of still images, and in 
addition, it is also possible to give several importance levels by purpose. Concretely 
speaking, for example, when a moving image is a drama, it is possible to give different 
importance levels, by a purpose of seeing an action scene, and by a purpose of desiring 
to see a scene which has someone in tear* respectively. 

[0075] 1005 designates an importance level selection button, and a name of an 
importance level, which is prepared for each button, is displayed. For example, a name 
of a user, a name by purpose such as an action scene and a scene which has someone in 
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tear, and so on are displayed. In order to carry out presentation of moving image 
through the use of another importance level, it is all right if the button 1005 is selected 
again. In the window 1003, when another importance level is selected, it is changed so 
as to display an importance level which corresponds to a button specified in response to 
it. 

[0076] As a kind of an importance level, it is also possible to dispose a personal history 
of a user. This is a thing for clearly showing a portion which a user already saw, and a 
portion which a user has not yet seen, in moving images, as an importance level. If a 
high importance level is .given to a portion which has not yet been seen, and if a low 
importance level is given to a portion which was already seen, respective classifications 
become obvious at a glance, by seeing a graph of an importance level. As a matter of 
course, it also becomes possible to select only a portion which has not yet been seen by 
adjusting the importance level threshold value setting lever 1000 and to present i. In 
this case, an importance level has to be updated every time a user sees a moving image. 
[0077] In addition to selecting one importance level out of a plurality of importance 
levels and utilizing it at the time of reproducing a moving image, it is also possible to 
select a plurality of importance levels, and define a new importance level by applying a 
predetermined calculation to the selected importance levels, and utilize it. For example, 
in case of selecting two importance levels, and selecting a portion in which an 
... importance level of one of them exceeds a threshold value set up by a. user, and .desiring- .. ... 
to display it, it is all right if a logical sum of two importance level is taken. More 
concretely speaking, it is assumed that, at the time that an importance level of "thrill" and 
an importance level of tc beauty" are prepared, it is desired to select and see only a scene 
which is thrilling, or in which beauty appears. Then, it is assumed that an importance 
level of "thrill" is (a) of Fig. 12, and an importance level of "beauty" is (b) of Fig.12. A 
user selects importance levels of 'thrill" and "beauty", and furthermore, instructs to carry 
out "presentation of a portion in which one of them exceeds a threshold value", and 
inputs a threshold value of an importance level. These importance level information are 
read out from the recording medium 107, when a user carried out selection of an 
importance level, or when a user selected source information such as moving image, 
sound, and group of still images, and stored in the importance level information 
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temporary storing section 100. Then, in order to carry out calculation of two 
importance level information, respective importance level information are sent to the 
modification calculating section 104* 

[0078] In the modification calculating section 104, in order to select and present a 
scene which is "thrilling", or in which tc beauty" appears, firstly, as to respective 
importance levels, 1 is allocated to a portion which exceeds a threshold value of an 
importance level inputted from the operation interface 109, and 0 is allocated to a 
portion which is less than it, and binarization is carried out {(c), (d) of Fig. 12). Then, a 
logical sum of binarized importance levels is taken with respect to each corresponding 
frame ((e) of Fig. 12). 

[0079] At this time, a frame for which 1 was obtained as a calculation result, is a frame 
to be presented. An importance level, which was newly prepared by calculation of two 
importance levels in this way, is sent to the importance level information temporary 
storing section 1 10. If necessary, the newly prepared importance level is recorded on 
the recording medium 107 through the recording control section 103. The new 
importance level, which was recorded in the importance level information temporary 
storing section 110, is sent to the presentation portion selecting portion in order to 
decide a portion which is actually presented to a user, and as a result of comparison with 
a threshold value, a presentation portion is decided. 

.[0080] As calculation between two importance levels, it is also possible to. use a logical . 
product, in addition to the above-mentioned logical sum. This can be utilized in case of 
desiring to see only a scene which exceeds a threshold value specified by both of the two 
importance levels. In the above-mentioned example, it can be utilized at the time of 
desiring to see a scene which is "thrilling" and in which "beauty" appears. In order to 
obtain a logical sum of two importance levels, two importance levels, which were sent to 
the modification calculating section 104, are compared with a threshold value in the same 
manner as described earlier, and binarization is carried out. Then, a logical product of 
the binarized importance levels are taken, and a new importance level is obtained, and it 
is sent to the importance level information temporary storing section 110. Subsequent 
processing is the same as the time of a logical sum. 

[0081] In addition, as another calculation, it is also possible to utilize average (sum) of 
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two importance levels. This is convenient in case that a user desires to utilize preferably 
an objective importance level, for example in case that a plurality of persons gave 
importance levels to the same image, and at the time of desiring to utilize an overall 
importance level to an image to which importance levels have been given by various 
purposes. In order to obtain an average importance level, firstly in the same matter as 
the time of carrying out a logical sum, two importance levels are sent from the 
importance level information temporary storing section 110 to the modification 
calculating section 104. In the modification calculating section, an average of two 
importance levels is calculated, and a new importance level is obtained. A new average 
importance level is sent back to the importance level information temporary storing 
section 11 0 9 and in the same manner as above, if necessary, it is recorded on the 
recording medium 107. Then, in order to decide a presentation portion, the new 
. average importance level is sent to the presentation portion selecting section, and by 
comparison with a threshold value inputted to the operation interface 109, selection of a 
presentation portion is carried out. 

[0082] A method of obtaining a new importance level by carrying out calculation 
between importance levels is not limited only to two importance levels. It is also 
possible to select much more importance levels and to obtain a new importance level by 
combining a plurality of calculations among them, and to utilize it. 

[0083] .In addition, in addition to presentation shown in Figs. 10. and. 11, it is designed 
so as to be able to select a method of presentation shown in Figs. 13 and 14. 
[0084] In Fig. 13, a setting lever 1000 of an importance level, and a time display 
window 1001 are the same as those in Figs. 10 and 1 1. However, it is different on such 
a point that, when the lever 1000 is set up, a representative frame is selected out of 
portions in which a given importance level exceeds a threshold value, and presented 
(1100), It is good to select representative frames by the number of sheets, which is 
sufficient to be able to display frames with high importance levels, but when it is 
designed in such a manner that a predetermined time width is disposed so as to prevent 
only frames, which are adjacent in terms of time, from being displayed, and only one 
frame is displayed in that time width, it is convenient since representative frames are 
selected from an entire moving image. In addition, when a threshold value of an 
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importance level is set up to be low, time of moving image to be presented becomes long, 
and the number of representative frames to be displayed becomes a lot, going with this. 
In the suchlike case, it is all right if it is designed to carry out display of representative 
frames across a plurality of pages. 

[0085] when a mouse cursor is pointed to a representative frame and clicked to specify 
it, a window 1 101, which displays a moving image, is displayed as shown in Fig. 15, and 
a moving image is reproduced from the specified frame. 

[0086] Display of Fig. 14 is a thing which displayed display of a representative frame as 
in Fig. 13 and a graph of an importance level simultaneously. Since it is clearly indicated 
about which portion of a moving image, a representative image corresponds to, it 
becomes apparent which portion of the moving image a user oneself will see by carrying 
out the suchlike display. 

[0087] In case of a group of still images, a still image, which was selected by an 
importance level, is displayed in the display window 1100, just all the same. In this 
regard, however, a horizontal axis of the importance level display window 1003 
represents an order of still images, but not time. In case of sounds, a bar chart, which 
shows sound wave forms and a status of ON/OFF of a specific sound, is displayed in the 
window 1 100, in the same manner as the display window 1 002. 

[0088] By the above-mentioned procedures, a user can present only a portion which is 
likely to be interesting in a moving image, selectively, .by utilizing, an importance level 
which has been already given. 

[0089] Next, user's procedures and presentation to a presenting section in case that a 
user gives an importance level to a moving image, a group of still images or sound, by a 
manual input, and in case of modifying an importance level which has been already given 
(different from the first embodiment), will be explained. 

[0090] Fig. 16 is an example of a screen which is presented on a presenting section, in 
case of carrying out giving an importance level to a moving image by a manual input, or 
of modifying an importance level which has been already given. 

[0091] Prior to input or modification of an importance level, an importance level, 
which is desired to input or modify, is selected by the importance level selection button 
1005. 1200 designates an importance level setting window, and in this, input or 
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modification of an importance level is carried out. 1006 is a thing which displayed a 
time portion displayed in the importance level setting window 1003, and an entire 
window is comparable to an entire moving image, and a shaded portion is comparable to 
a time portion which is displayed in the window 1002. 1201 designates a display frame 
position mark for displaying which portion, a frame currently displayed in the 1002 is. 
[0092] As to the importance level setting window 1200, time of a moving image 
corresponds to a longitudinal direction, and a scale of time is displayed on an upper part 
of the importance level setting window. When a mouse cursor 1007 enters in the 
importance level setting window, a frame of time, which corresponds to a position of the 
mouse cursor, is automatically displayed in 1002. In addition, when the mouse cursor 
gets out from an inside of the importance level setting window .1200, an image at the 
time, which corresponds to a cursor position at such a moment that the mouse cursor 
gets out, is continuously displayed. At the time of carrying out editing of an importance 
level given to a group of still images, in the same manner as the occasion of reproducing 
a group of still images, a horizontal axis of the importance level setting window 1200 
represents an order of what number of still images. When a mouse cursor exists in the 
importance level setting window, a still image of number which corresponds to a position 
of the mouse cursor in a horizontal direction, is always displayed on the display window 
1002. On the one hand, at the time of carrying out editing of an importance level given 
to sound,. centering around time which corresponds to apposition. .of the cursor in a 
horizontal direction, wave forms in a zone across its vicinity are displayed in the 
importance level setting window 1200. It is all right even if it is not display of wave 
forms, but display of a bar chart showing ON/OFF of specific sound. Then, when a 
mouse cursor is moved in the importance level setting window 1200, a sound at 
corresponding time is outputted in tune with a speed of the movement. If the mouse 
cursor is moved slowly, unhurried sound is outputted, and if it is moved quickly, fast 
sound is outputted. At this time, high and low of sound is changed by such a speed that 
the mouse cursor was moved, and therefore, it is all right even if processing of 
suppressing this is carried out. In order to carry out setting of an importance level, 
operations shown in Figs. 17 through 19 are carried out, and they will be explained 
below. 
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[0093] When an importance level has not been given yet, an importance level of a 
reference revel is given to all portions. Therefore, importance level display in the 
importance level setting window is a flat one of only reference leveL Firstly, a mouse 
cursor is moved in the importance level setting window so as to display a first frame of a 
portion to which an importance level is desired to be given. When the first frame is 
displayed on the display window of 1002, a mouse button is pushed (Fig. 17). The 
mouse cursor is moved over keeping to push the mouse button (drag), and the mouse 
cursor is moved so as to display a last frame to which an importance level is desired to be 
given. At this time, as to movement of the mouse cursor in the importance level setting 
window, a longitudinal direction corresponds to time of a moving image, and an up and 
down direction corresponds to an importance level. Therefore, over moving the mouse 
cursor from side to side so as to display a last frame of a portion to which an importance 
level is given, the mouse cursor is also moved in up and down direction, and pointed to a 
desired importance level (Fig. 18). if display of a target frame and an importance level 
which is desired to be given, are obtained simultaneously, the mouse button is released 
(Fig. 19). by the above-mentioned operation, an importance level, which was flat at first, 
is changed by a portion specified as shown in Fig. 19. 

[0094] Values of the same importance level are given to a portion to which a plurality 
of importance levels were set by one operation. In addition, here, a first frame of a 
.zonela which an importance level is desired to be given T was-firstly specified,. and then, a 
last frame was specified by the mouse, but it is also possible to specify a last frame of a 
zone to which an importance level is desired to be given, and then, to specify a first 
frame and an importance level simultaneously. 

[0095] In Figs, 17 through 19, an importance level was set to above a reference level, 
but it is also possible to specify an importance level which is on a lower side than the 
reference level, and to set up a negative importance level. In case that a portion, which 
is not desired to be displayed so much, is included in a moving image, it is all right if a 
negative importance level is set up. In order to give an importance level to a portion to 
which an importance level has not been given, it is all right if the following operation is 
carried out repeatedly. 

[0096] When modification is applied to an importance level which was once set up (an 
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importance level in a specified range is enlarged/reduced by given magnification), a 
mouse cursor is firstly pointed to a first portion of a portion which is desired to be 
modified, in the importance level setting window. After it is confirmed that a frame, 
which is displayed in the display window 1002, is a first frame of a modification starting 
portion, a mouse button is released (Fig.21, Fig>21). When the mouse cursor is moved 
over keeping to push the mouse button, a range of modifying an importance level is 
specified by movement in a longitudinal direction, and an enlargement/reduction level of 
an importance level in a change range is specified by movement in an up and down 
direction (Fig.22). At this time, magnification of enlargement/reduction is set up in 
such a manner that a maximum importance level in the change range corresponds to a 
position of the mouse cursor in an up and down direction. That is, when the highest 
value of an importance level in a change zone is Imax, and an importance level, which 
corresponds to a position of the mouse cursor in an up and down direction, is Ic, all of 
importance levels in the change range are magnified by lc/Imax times. When a change 
range and a change value are decided, the mouse button is released, to fix the change 
(Fig.23). At this time, if the mouse cursor is pointed to a lower position than a 
reference value of an importance level with keeping to drag the mouse and fixing is 
carried out, it is possible to change a positive importance level in the change range to a 
negative importance level. As a matter of course, it is also possible to change a 
negative importance level to posative.by the same operation ((a) -_4c)_of Fig.24). ... 
[0097] In addition, finally, if the mouse cursor is pointed to a position of a reference 
level and the mouse button is released, the change range is all set up again to the 
reference level, and therefore, it is possible to utilize it on the occasion of canceling an 
importance level. 

[0098] In the above-described explanation, a procedure of newly setting an importance 
level, and a procedure of modifying an importance level which has been set up, are the 
same. As to which one of newly setting and modification is carried out, it is decided by 
whether or not an importance level, which has been already set up, exists in a time range 
specified by dragging and moving a mouse from side to side. If all of importance levels 
in the specified time range are the reference level, it becomes setting of a new importance 
level, and if an importance level other than the reference level is included in the specified 
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time range, modification is carried out. 

[0099] In addition, setting and modification of an importance level can be carried out 
only in a time range displayed in the importance level setting window 1200. In order to 
change a time range in the importance level setting window, a display range specifying 
window 1006 is used (Figs.25 - 27), Entirety of the display range specifying window 
represents an entire moving image, and a shaded portion in it is a time range which is 
displayed in the importance level setting window. On an upper part of the display range 
specifying window, corresponding time is displayed. In order to change a display time 
range in the importance level setting window, a mouse cursor is pointed to a shaded 
portion in the display range specifying window, and moved from side to side over 
keeping to drag it, and a mouse button is released at an appropriate location, to fix a 
destination of movement. When the shaded portion is moved in this way, it is possible 
to move the display time range without changing a length of a range to be displayed. 
On the one hand, when a head, or dead end portion of the shaded portion is dragged and 
moved, the dragged head, or dead end portion is moved together with a mouse cursor. 
At this time, a size of the shaded portion in the display range specifying window changes 
along with movement of a mouse cursor, and therefore* a length of the time range, which 
is displayed in the importance level setting window, also changes. 

[0100] A change of a time range (in case of a still image group, it becomes a range of 
an order,.from what number. to what number), displayed in. the importance level, setting 
window, is carried out in the display range specifying window, and can be carried out 
also by clicking a button 1202 and 1203 of Fig. 12 through the use of a mouse. 1202 
designates a scroll button in a longitudinal direction, and by pointing a mouse cursor to 
this button and pushing a mouse button, a time range, which is displayed in the 
importance level setting window, is moved in a direction which is displayed on the button. 
Going with movement of the display time range, a shaded portion in the display range 
specifying window 1006 also moves. Fig.25 is a display example at the time of 
pushing a rightward (forward direction of time) scroll button. When scroll is carried 
out, the display frame position mark 1202 does not move to the importance level 
specifying window. Therefore, in the course of pushing a scroll button to carry out 
scroll, a frame, which is displayed in the image display window 1002, is changed every 
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second according to time pointed out by the display frame position mark 12201. 
Therefore, it is desirable to dispose a scroll button at a right end and a left end of the 
importance level setting window as shown in Fig. 16. Because, firstly on the occasion 
of scrolling rightward, if an image at a right end in the importance level display window 
is displayed, it is easy to decide to what extent, scroll should be canied out. Then, if a 
right scroll button is set up to a right end of the importance level setting window, a 
position where a mouse cursor gets out of an inside of the importance level setting 
window becomes a position close to the right end in the course of nature. As its result, 
the display frame position mark is fixed to the right end, and a frame to be displayed 
becomes a right end portion of the importance level setting window. It is desirable that 
a leftward (direction of going back in time) scroll button is located at a left end of the 
importance level setting window for the same reason. 

[0101] 1203 designates a magnification change button of the display time range,' and 
when a button, on which "2" is displayed, is clicked, a derail level of display in a time 
zone, which is displayed in the importance level setting window, becomes double. That 
is, a range, which is displayed in the importance level setting window, becomes half, but 
it is displayed larger, and in more detail, by that much. At this time, a frame, which 
corresponds to the display frame position mark 1201, is displayed in such a manner that 
its position in the importance level setting window does not change. Fig.26 is a display 
example prior. to pushing the. magnification change button. 1203^ .and. after. "2? of. the. 
magnification button is pushed, it is changed to display of Fig,27. After "2" of the 
magnification button is pushed, a display time range of the display frame position mark in 
a longitudinal direction becomes half "1/2" of the display magnification change button 
1203 is a button which reduces a detail level of display to half and doubles a time range 
to be displayed, in a direction opposite to "2". Just all the same, when magnification is 
changed^, a frame of the display frame position mark is displayed in such a manner that its 
position in the importance level setting window does not change. In addition, as to 
either one of display magnification changes, after a change is instructed, a size of a 
shaded portion in the display position specifying window 1006 becomes double or 1/2. 
It is desirable to dispose the display magnification change button 1203 in the vicinity of a 
center of the importance level setting window 1200. This is based on the same reason 
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as arrangement of the scroll button. That is, on the occasion of changing magnification 
of display, there are many cases to desire to display a center portion in the importance 
level setting window 1200 even after magnification change. On that account, there is 
need to dispose the display frame position mark at a center portion in the importance 
level setting window 1200. Therefore, a position where a mouse cursor gets out of an 
inside of the importance level setting window 1200 has to be set to a center portion, and 
after all, if the magnification change button 1203 is set up to a center portion, it becomes 
possible to cany out the suchlike operation in the course of nature, 
[0102] Meanwhile, in the same embodiment, magnification which can be changed, is 
only two kinds of double and 1/2 times, but it is all right even if much more 
magnifications are prepared, and it is designed in such a manner that a user can specify 
magnification. 

[0103] If selection of an importance level, newly setting, change, and change of a range 
to which an importance level is given, are carried out as described above, it is possible to 
carry out setting of an importance level to moving image and sound, a group of still 
images. 

[0104] Next, a preview operation, in which after an importance level is given once, it is 
confirmed whether there is a change once more, will be explained. 

[0105] An operation of confirming whether an appropriate level was given or not after 
an. importance. leveL was .given is. preview. The preview is carried out. under display, 
shown in Fig.28. Display of Fig.28 is the same as display of Fig. 1 1 except a portion of 
a right side. Therefore, if the same portion as in Fig. 11 is utilized, it is possible to carry 
out exactly the same operation as normal reproduction. Here, threshold value setting of 
different importance levels is carried out several times, and reproduction of a moving 
image is carried out, and if setting of importance levels is unsatisfactory, setting of an 
importance level may be terminated without making a modification. 
[0106] However, there is such a case that need to carry out a modification arises when 
a portion to which an unsatisfactory importance level is given, and a portion to which a 
modification is desired to be made, are found several. In the suchlike case, a marking 
button 1300 and a jump button 1301 are used. The marking button is useful in case 
that a portion, which is desired to be modified, is marked and they will be collectively 
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modified at later time. Reproduction is carried out by preview, and very time a portion, 
which is desired to be modified, is found, the marking button 1300 is pushed. When 
reproduction by preview is completed roughly as a whole, it goes back again to display 
of a screen for carrying out giving, modification of an importance level by manual input 
(Fig.29). At this time, at a marked position, a mark 1302, which shows marking, is 
displayed. Therefore, it is all right if this portion is find out and modification of an 
importance level is carried out. An operation of modification is as described above, but 
it is designed in such a manner that, when an importance level at a marked position is 
changed, marking disappears automatically. In addition, when a marked position exists 
in a moving image, the jump button 1303 to marking is displayed. When this button is 
pushed, a marked portion is searched in a earlier portion in terms of time, than a frame 
position which is currently displayed, and a time range in the importance level specifying 
window is changed in such a manner that the portion is displayed. By use of this button, 
it is jumped to a marking portion one after another, and modification can be carried out 
effectively. 

[0107] The jump button 1301 is utilized in case of desiring to carry out modification of 
an importance level immediately during a period of preview. When the jump button is 
pushed during a period of preview, display of giving, modification of an importance level 
by manual input is carried out immediately. A range of an inside of the importance level 
setting, window.is. set up. so as to center on a frame which is -displayed by. preview when 
the jump button is pushed. Therefore, it is possible to immediately modify an 
importance level in the vicinity of a frame when the jump button is pushed. 
[01 OS] The preview and modification described above are repeated, and if a portion to 
be modified disappears, an importance level giving/modifying work is completed. 
[0109] (Third Embodiment) Next, as a method of estimating an importance level in an 
information processing apparatus which relates to the present invention, an example of 
utilizing eye movement observation will be explained with reference to drawing. 
[0110] fig.30 is a graph which shows a method of estimating an importance level from 
an appearance of visual point movement in an information processing apparatus relating 
to a third embodiment. 

[0111] A process of importance level estimation, which will be explained below, is a 
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portion which is comparable to internal processing of the judgment section 102 explained 
in the above-described first embodiment. Output data, which is outputted from an eye 
movement measuring device, is converted into visual point data which a user is actually 
seeing on a screen by executing appropriate amendment and calculation processing. A 
process of outputting visual point data from an eye movement measuring device may be 
an arbitrary method. As an eye movement measuring device, as of now, one of 
irradiating eye balls with infrared ray and observing a reflected light strength from a 
sclerotic coat, one of placing electrodes in the vicinity of eyes and observing an electric 
potential difference in eye balls, one of mounting a contact lens incorporated with an 
electromagnet and detecting a magnetic field, or one of taking an image of eyes and 
calculating a visual point by image processing, and so on, have been known. 
[0112] A time difference (differentiation) of visual point data obtained as above is 
taken, and a visual point moving speed every time is obtained. Saccade, which is the 
fastest movement among human's eye ball movements, occurs mainly at the time of 
moving a visual object during a period of 20 - SOrmllsecconds, but in the information 
processing apparatus of the same embodiment, a portion, in which the visual point 
moving speed 901 exceeds a threshold value 902, is qualified as a saccade period 903. 
It has been known that resolution of an input to visual sense is low, because of a fast eye 
ball movement during a period of saccade, but it has been clarified that an unclear image 
is realized. during a. period from, several dozen milliseconds before saccade. (904) up. to 
approximately 200 milliseconds after saccade (905). Therefore, it is possible to assume 
that an amount of information, which a user obtains from a screen, is few, during a series 
of periods; a period of before saccade 904, during a saccade period 903, and during a 
period of after saccade 905. Therefore, to a period 906 in which it is desired to obtain 
an importance level, it is possible to use a ratio of a total of "time that a user obtained 
information effectively from a screen", i.e., high information density time 907, to an 
importance level measurement period 906, as an index of a height of an importance level. 
The importance level measurement period 906 was explained, in the first embodiment, as 
5 second unit, cut unit, etc. Here, the threshold value 902 is a value of approximately 
10-50 [decrees/second]. 

[0113] Fig.31 shows a result of an experimentation in which an importance level was 
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estimated from sight line analysis by the method of the same embodiment. In this 
experimentation, 1 5 sheets of still images are presented to a person being tested, and the 
person is allowed to see only for free time every 1 sheet. Therefore, it is all right even if 
time until a person being tested completes an inspection is used as one scale of evaluating 
a height of an importance level of a person being tested oneself In the figure, a 
horizontal axis is inspection time, and a vertical axis is an importance level estimated by a 
method of the present invention. A correlation of approximately 80% was seen 
between both sides. If saccade occurs at a given frequency regardless of an image 
content, an importance level becomes constant since the number of saccades also 
increases, as compared to presentation time. In sum, in the figure, points should gets 
lined up on a horizontal line, but as seen in Fig.3 1, a relatively high importance level has 
been estimated for an image which was seen long (= it is conceivable that an importance 
level was high), and it backs up possibility of the method of the same embodiment. 
[0114] By the foregoing, it is possible to estimate an importance level of a user, from 
an eye ball movement which is an involuntary and natural reaction of a user on the 
occasion of seeing an image. 
[0115] 

[Advantage of the Invention] As described above in detail, according to an information 
processing apparatus which relates to the present invention, it becomes possible to give, 
modify additional information,-e.g. 9 an importance level to multimedia information,. and it. 
can be realized in such a form that feature extraction of information by use of the 
importance level reflects an intention of a user, and without requiring advanced 
knowledge processing. 

[0116] In this way, feature extraction of information becomes easy, and thereby, 
shortening of user's access time to information becomes possible, and it contributes to 
burden reduction to a network, and it is possible to provide an interface for information 
access which does not press a user for specialized knowledge and which is easy to use. 
[Brief Description of the Drawings] 

[Fig. 1] is a block diagram which shows a configuration of an information processing 
apparatus relating to an embodiment of the present invention. 

[Fig.2] is a flow chart which shows algorithm of modifying an importance level of an 
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information processing apparatus relating to a first embodiment 

[Fig.3] is a conceptual view for explaining a method of estimating a change of an 
importance level in an overflow zone of the information processing apparatus relating to 
the first embodiment, 

[Fig.4] is a conceptual view for explaining an operation method on the occasion of 
carrying out change/modification of an importance level of the information processing 
apparatus relating to the first embodiment. 

pFig.5] is a conceptual view for explaining a change procedure method at the time that an 
importance level of a change destination of the information processing apparatus relating 
to the first embodiment was decided. 

[Fig. 6] is a conceptual view for explaining a method of processing/presentation which 
does not utilize a semantic separation of information of the information processing 
apparatus relating to the first embodiment. 

[Fig.7] is a conceptual view for explaining a method of processing/presentation which 
utilizes a semantic separation of information of the information processing apparatus 
relating to the first embodiment. 

[Fig, 8] is a conceptual view for explaining an operation principle of an address 
generating section on the occasion of carrying out "sequential presentation" of the 
information processing apparatus relating to the first embodiment. 

[Fig.9] is a conceptual, view., for explaining an operation principle of such, a case. that, 
information of the information processing apparatus relating to the first embodiment is 
music (tune) and a separation of information has been known. 

[Fig. 10] is a view which shows a display example of a presenting section of an 

information processing apparatus relating to a second embodiment. 

[Fig. 11] is a view which shows a display example of the presenting section of the 

information processing apparatus relating to the second embodiment. 

[Fig. 12] is a conceptual view for explaining a processing method utilizing a plurality of 

importance levels of the information processing apparatus relating to the second 

embodiment. 

[Fig. 13] is a view which shows a display example of the presenting section of the 
information processing apparatus relating to the second embodiment. 
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[Fig. 14] is a view which shows a display example of the presenting section of the 

information processing apparatus relating to the second embodiment. 

[Fig. 15] is a view which shows a display example of the presenting section of the 

information processing apparatus relating to the second embodiment. 

[Fig. 16] is a view which shows a display example of the presenting section of the 

information processing apparatus relating to the second embodiment. 

[Fig. 17] is a conceptual view for explaining an operation method on the occasion of 

setting an importance level of the information processing apparatus relating to the second 

embodiment. 

[Fig. 18] is a conceptual view for explaining an operation method on the occasion of 
setting an importance level of the information processing , apparatus relating to the second 
embodiment. 

[Fig. 19] is a conceptual view for explaining an operation method on the occasion of 
setting an importance level of the information processing apparatus relating to the second 
embodiment. 

[Fig.20] is a conceptual view for explaining an operation method on the occasion of 
making a modification to an importance level of the information processing apparatus 
relating to the second embodiment. 

[Fig,21] is a conceptual view for explaining an operation method on the occasion of 
making a modification .to . an .importance level of the information processing apparatus . 
relating to the second embodiment. 

[Fig.22] is a conceptual view for explaining an operation method on the occasion of 
making a modification to an importance level of the information processing apparatus 
relating to the second embodiment. 

[Fig.23] is a conceptual view for explaining an operation method on the occasion of 
making a modification to an importance level of the information processing apparatus 
relating to the second embodiment. 

[Fig.24] is a conceptual view for explaining an operation method on the occasion of 
making a modification to an importance level of the information processing apparatus 
relating to the second embodiment. 

[Fig .25] is a conceptual view for explaining an operation method on the occasion of 
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carrying out setting and modification of an importance level of the information 
processing apparatus relating to the second embodiment. 

[Fig.26] is a conceptual view for explaining an operation method on the occasion of 
carrying out setting and modification of an importance level of the information 
processing apparatus relating to the second embodiment. 

[Fig.27] is a conceptual view for explaining an operation method on the occasion of 
carrying out setting and modification of an importance level of the information 
processing apparatus relating to the second embodiment. 

[Fig.28] is a conceptual view for, explaining a preview operation of the information 
processing apparatus relating to the second embodiment. 

[Fig.29] is. a conceptual view for explaining a preview operation of the information 
processing apparatus relating to the second embodiment. 

[Fig. 30] is a graph which shows a method of estimating an importance level from an 
appearance of a visual point movement of an information processing apparatus relating 
to a third embodiment. 

[Fig.31] is a view which shows a result of an experimentation in which an importance 
level was estimated from sight line analysis by a method shown in the third embodiment. 
[Description of Reference Numerals and Signs] 

101.. .detector, 102... judgment section, 103... recording control section, 
104...modification . calculating, section, . 105. ..reading control., section, 106_,.iraage 
presenting section, 107... recording medium, 108... input device, 109... operation interface, 
110... importance level information temporary storing section, 111... presentation portion 
selecting section, 1 12.. .address generating section, 113. ..original information temporary 
storing section, 11 4... image generating section, 11 5... sound generating section, 
116.. .sound presenting section, 301. ..overflow zone, 302,. .entrance of overflow zone, 
303. ..exit of overflow zone, 304...importance level to be estimated, 402.. .screen in which 
maximum value of importance level was recorded, 403. ..screen by which order change is 
desired, 405„.position of order change destination, 601... threshold value, 603. ..time 
transition of importance level, 604.„time zone to be selected, 605. ..maximum value in 
time zone to be selected, 702. ..scene change, 703. ..cut, 704.. .maximum value every cut, 
705... threshold value, 706... importance level density, 802... phrase in which importance 
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level is high, 902„ .threshold value of saccade, 904... low information period before 
saccade, 905... low information period after saccade. 
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[Fig-1] 
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[Fig. 18] 
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[Fig.6], [Fig.7], [Fig.9] 
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[Fig. 17] 
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